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ABSTRACT 

The involvement of an actual class in the conduct of 
a product evaluation is described as a powerful learning experience 
for the students and a valuable service for a client. The essential 
elements of such a program are: (1) a teacher-negotiated contract for 
an evaluation; (2) students who sign on by enrolling for credit; (3) 
background provided by the client; (A) formation of teams of 
classmates to conduct the work; and (5) reports of evaluation results 
to the client. The practice of paying the class for evaluation work 
is advocated as an incentive. Two specific evaluations by graduate 
students at Utah State University undertaken in this manner, one 
evaluating an approach to teaching French or Spanish at the 
elementary level and one evaluating college-level instructional 
materials for a distance learning course in French, are described. 
How this evaluation approach relates to the philosophy of John Dewey 
is explored, and the operation of the concepts of experiential 
learning is discussed. (Contains 12 references.) (SLD) 
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Introduction: 



The conduct of evaluation can be viewed as essentially a craft skill, 
with a need for sustained learning and supervision prior to and 
during the process of work. Others teachers in the field (e.g. 
Morris, 1989) have argued that experiential learning, where the 
learner profits from direct experience, is a powerful teacher. The 
volume by Mertens (1989) contains numerous examples of those 
teaching evaluation using direct experience as a teacher (e.g. 
Preskill, 1989). There seems to be general agreement that learning 
by direct experience is an ideal way to structure the training of 
evaluators, if such situations can be devised practically. 

In an earlier article (Eastmond, Saunders & Merrell, 1989), I argued 
that the involvement of an actual class in the conducting of an 
evaluation could provide a powerful learning situation for students. 
The essential elements of this kind of learning are (1) the teacher 
negotiates a contract for a certain evaluation; (2) students n sign 
on" by enrolling in the course for credit, with the understanding 
that they will be paid for their work (generally the off-campus field 
work portion); (3) the client meets with the class — in person or via 
electronic distance education means — to provide the background 
\ on the evaluation problem, as well as the expectation that this 

vQ project is "for real"; (4) class members work in teams to conduct 

(s) the work, learning how to perform their tasks as they go; and (5) 

^ the class reports back to the client, in oral and in written form, to 

V provide the client with the evaluation results. This process provides 

a powerful learning experience for students and a valuable service 
for a client. 
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Three caveats are in order in conducting such a project. First, the 
client must be aware of the nature of the project as part of the 
contract and must be willing to meet and work with students in this 
way. Second, the students must be prepared for the necessary tasks 
and encouraged to do work for which they inevitably feel 
inadequate, with the teacher taking final responsibility to edit and 
ensure the quality of any product. And finally, the project must be 
managed carefully to see that the work is completed when the 
school term is finished, so as not to leave the teacher "holding the 
bag." 

Paying students for their work in completing a recognized degree is 
frequently done in graduate school, although usually on an 
individual basis and as some form of research or teaching 
assistantship. The practice of paying an e itire class for their work 
on such a project is less common; however, in this evaluator f s 
experience it is welcomed by students and generally agreeable to 
both clients and administrators once the nature of the service to be 
performed is made clear. Once the word of it gets out to students, 
student enthusiasm certainly can boost enrollment. In the classes 
under consideration, payment of $75 to $150 per student for their 
work on the final has been about normal, though the summer 
course paid $25 per student. The policy formally accepted by the 
Research and Evaluation Group at Utah State University is that, "if 
the professor receives payment for work done by a class, then the 
students should be paid as well." 

The Specific Experiment: 

In 1993, two specific evaluations were undertaken by graduate 
students in the Instructional Technology 679: Instructional Product 
Evaluation, both involving teaching of foreign language. The first 
(Winter 1993, enrollment 17) evaluated the effectiveness of a novel 
approach to teaching Spanish or French at the elementary school 
level, using a videodisc with hand-held barcode reader and 
accompanying teacher manual and student worksheets. The 
approach was innovative in that it required no prior language on the 
part of the foreign language teachers, but rather, they were to learn 
along with the students. The second instance of the same class 
(Summer 1993, enrollment 18) was the evaluation of the 
instructional materials for a distance education course in 
introductory French, using audiotape, course manual and computer 
review exercises. This college level course is currently being taught 



in three rural high schools in Utah, where students work 
independently and submit their work to a professor of French at 
Utah State University, The two resulting reports have been 
published as local evaluation reports (Eastmond, Durrant, & 
Samhouri , 1993; and Eastmond and Elwell, 1993). 

A perspective from John Dewey: 

An early advocate of experiential learning was Dr. John Dewey, a 
foremost American philosopher of his day whose influence reaches 
into our own. With only a bit of hyperbole, British philosopher A. N. 
Whitehead has termed Dewey "the chief intellectual force providing 
[the North American] environment with coherent purpose." 
(Johnson, 1949, dust jacket). Dewey was particularly drawn to the 
problems of education, and his educational philosophy constitutes a 
vital portion of his total system of philosophy. 

Given the extensive work completed in his 93 year lifetime, it is hard 
to represent Dewey's positions briefly without distorting them. The 
following will attempt to draw upon direct quotations where 
possible and to give the reader the flavor of Dewey f s robust thought. 

Dewey believed that education was essentially the "reconstruction of 
experience", in helping the person learn to value and learn from 
happenings in the environment (Brumbaugh & Lawrence, 1963). 
Dewey taught that the mind and body must both be engaged, and 
that the dualistic theory that would attempt to separate them, 
labeling "mind" as the hero to be cultivated and "body" the culprit 
to be controlled, was doomed to failure when applied to real boys 
and girls. He felt that the wise teacher took a holistic view of the 
person in structuring learning experiences. The following 
quotations attest to the powerful nature of educational experience: 

An ounce of experience is better than a ton of theory, simply 
because it is only in experience that any theory has vital and 
verifiable meaning* 

An experience, a very humble experience, is capable of 
generating and carrying any amount of theory (or intellectual 
content, but a theory apart from experience cannot be 
definitely grasped even as theory. It tends to become a mere 
verbal formula, a set of catchwords used to render thinking, 



or genuine theorizing, unnecessary and impossible. (Dewey, 
1944, p. 144). 

In an evaluation context, premature emphasis upon verbal (or 
pictorial) models of evaluation in the absence of concrete 
experience becomes scholastic and formal. The person who learns 
formally without direct experience may be able to supply verbal 
description, but will be unlikely to have the ability (or the 
confidence) to proceed directly to the completion of an actual 
evaluation problem. Only when we can ground this knowledge in 
real experience to see the interplay of expectations, constraints, 
and political forces — can the student obtain any idea of the 
complexity of the evaluation enterprise. The direct experience with 
an evaluation client is a liberating influence, while theorizing in the 
abstract is more likely to arrest growth and confidence than to 
promote it. 

Dewey frequently makes use of a version of the scientific method. 
Indeed, for Dewey, the scientific method exemplified how problem 
solving occurred, and this kind of problem solving formed the basis 
of all reflective thinking. Taking Dewey's problem solving stages 
and matching them against the actual experience of a class 
conducting an evaluation demonstrates the mental effort required 
to succeed at this enterprise: 

General features of a reflective experience (Dewey, 1944, p. 150) 
are followed by observations of a real class working through an 
evaluation contract: 

1. Perplexity, confusion, doubt, due to the fact that one is 
implicated in an incomplete situation whose full character is not yet 
determined. 

Initial enthusiasm of students quickly gives way to feelings of 
inadequacy: "You expect us to conduct an evaluation when this is 
our first exposure to 'hese concepts?" "How can I expect a client to 
pay me for work that I am just learning how to do?" 

2. A conjectural anticipation - a tentative interpretation of the 
given elements, attributing to them a tendency to affect certain 
consequences; 
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— Trying to figure out the real motives of the client. "Is he really 
ready to make revisions if our evaluation shows it is necessary?" 
"Who is in control in this setting?" "Does their inservice program 
really work?" 

3. A careful survey (examination, inspection, exploration, analysis) 
of all attainable consideration which will define and clarify the 
problem in hand; 

— Students working to define parameters and constraints: "What 
are our limits in terms of budget and timelines?" (And in personal 
terms:) "How will I be graded on this work?" 

4. A consequent elaboration of the tentative hypothesis to make it 
more precise and more consistent, because squaring with a wider 
range of facts; and . . . 

Defining the research (or evaluation) questions: "And so these 
are the questions our team's evaluation should focus on, right?" 
"The client says this is a formative evaluation, but aren't they really 
asking for summative information? What if we get to the end and 
can't answer questions about overall effectiveness? What then?" 

5. Taking one stand upon the projected hypothesis as a plan of 
action which is applied to the existing state of affairs: doing 
something overtly to bring about the anticipated result v and thereby 
testing the consequences. 

— Moving ahead to gather data and draw conclusions, based upon a 
working hypothesis: "O.K. Let's assume that (our client) really 
wants to know if this program works. Here is what we have to do to 
find out." "Based upon these surveys of participants, as well as our 
use of the materials, we can conclude that these features of the 
program are working well, and these (others) are not." 

The point is that having a real evaluation contract raises the stakes 
of these steps. Knowing that someone will make use of the results, 
and is paying to have them, helps the student to understand the 
seriousness of this training. Having to verbally report back to the 
client and to justify any conclusions drawn simply reinforces that 
sense of purpose. 



And from a teaching standpoint, the point of this exercise is not 
simply the fulfilling of a contractual obligation. It is the cultivation 
of mental skills in students, the skills necessary to take on a 
contract and to perform a worthy evaluation. For many of these 
students, it is the first time that they have tried to obtain a fee for 
their services in any sort of consulting arrangement. The notions of 
written contract, a daily rate, managing activities and budget, and 
providing deliverables to a client are all rather new concepts for all 
but a few of the students. The experience plunges them into a new 
environment where they have to "sink or swim 1 '. With a little 
coaching, they invariably "swim", even the most reluctant or modest 
in their abilities. And in doing so, they experience an intellectual 
exercise of the first order. 

In Conclusion: Dewey is quite explicit in his praise for "education 
through occupations 1 ', in some settings as "vocational education", 
and the kind of experience the contractual evaluation exercise 
provides: 

An occupation is a continuous activity having a purpose. 
Education through occupations consequently combines within 
itself more of the factors conducive to learning than any other 
method. It calls instincts and habits into play; it is a foe to 
passive receptivity. It has an end in view; results are to be 
accomplished. . . (O)bservation and ingenuity are required at 
each stage to overcome obstacles and to discover and readapt 
means of execution (p. 309). 

The only adequate training for occupations is training through 
occupations (p. 310). 

Dewey would, as much as possible, remove the line between work 
and other activities. He makes a point that such a change must 
happen gradually, however: 

As we have seen (in an earlier chapter) in older pupils work is 
to educative development of raw native activities what play is 
for younger pupils. Moreover, the passage from play to work 
should be gradual, not involving a radical change of attitude 
but carrying into work the elements of play, plus continuous 
reorganization in behalf of greater control (p. 315). 



Even though the evaluation class under consideration is part of a 
graduate program leading to a professional specialization as an 
instructional technologist, one criticism of the program could be 
that the transition into the world of work is too abrupt. Because 
other clauses in the program seldom engage in contractual activities 
with an outside client and because the work must be undertaken 
under time constraints of no more than ten weeks, the change can 
be an abrupt one for students. Some relish it and cite it as the high 
point of their coursework in the department; others resist it for a 
variety of reasons, the main one being that the experience is so 
different from their other coursework and may be something quite 
different than what they had expected in enrolling in the course. 
Some resist the lack of structure and feel that excessive demands 
are being put on their time ("Welcome to graduate school!"). 

Student course evaluations are generally positive, but not 
overwhelmingly so. Because each evaluation contract has a specific 
set of demands and challenges, the work lacks the relaxed feel and 
predictability of some of the students 1 other coursework. For 
example, in the evaluation of the Spanish/French videodisc program 
"Hablar et Parler", students received the course outline in three 
installments, since it was impossible to predict the quarter's 
activities with accuracy. And it may be unrealistic to keep the 
majority of other class assignments when an evaluation contract is 
underway. The class taught summer quarter was conducted in 
exactly four weeks, and student complaints about workloads 
reached a new high. They completed the work, satisfied the 
customer, and grew from the experience, but some felt that quality 
had been sacrificed to quantity and that the experience had been 
just too intense. Some of the assigned work may be consolidated in 
the future, but only with some concern by the instructor. The in- 
class learning must not supplant the contract experience, but rather 
should be a supplement to it. Students need to read the text 
material to obtain the theory to back the actual work they are 
conducting. 

But these are the considerations one wrestles with as an instructor 
in a demanding class. In any case, the value of the contractual 
work, in Dewey's terms, must be in reflective practice. Schon 
(1983, 1987), who describes such work as the ideal for training in 
the professions, appears to have gained immensely from Dewey. 
The class members must grapple with real issues, in groups or on 
their own, to devise unique solutions to the problem posed by 
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evaluating a program or product. If they can be paid for their 
work, so much the better. But in any case, the framework provided 
by Dewey will be of help in prescribing the kind of thinking one 
could expect in this situation and why its effects are so powerful. 

For reprints or further information, contact: 



Dr. Nick Eastmond, Professor 
Dept. of Instructional Technology 
Utah State University 
Logan, UT 84322-2830 
Tel. (801) 750-2642 

Internet: NEAST@CC.USU.EDU 
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